As the use of digital camcorders grows, to capture videos using hand-held camcorders becomes more and more convenient than before. However, since most people usually do not bring a tripod with their camcorders, unwanted vibration in video sequence is an unavoidable effect due to the handshakes. To avoid or remove the annoying shaky motion is one of the significant problems in home videos, and video stabilization is an important technique to solve this problem. Many existed video stabilization applications result a stabilized video by smoothing the camcorder motion path and then truncating the missing areas after aligning the video frames along the smoothed motion path. Hence, the stabilized videos still have many unexpected movements, since only high frequency shaky motions are removed during the smoothing stage. Moreover, the video qualities of the stabilized videos are usually decreased due to the truncating stage.
Video stabilization is an important research topic in the fields of multimedia, image processing, computer vision, and computer graphics. Buehler et al. (“Non-Metric Image-Based Rendering for Video Stabilization,” Proc. IEEE CVPR 2001, Vol. 2, pp. 609-614, 2001) proposed an image-based rendering (IBR) method to stabilize videos. Recently, image processing methods are widely used for video stabilization. For estimating the camcorder motion path, Litvin et al. (“Probabilistic Video Stabilization using Kalman Filtering and Mosaicking,” Proc. SPIE EI 2003, Vol. 5022, pp. 663-674, 2003) estimated a new camcorder motion path by altering camera parameter, and Matsushita et al. (“Full-Frame Video Stabilization,” Proc. PS IEEE CVPR 2005, Vol. 1, pp. 50-57, 2005) smoothed the camcorder motion path to reduce the high frequency shaky motions. However, although the high frequency shaky motions can be easily reduced, the stabilized videos still have low frequency unexpected movements. Gleicher and Liu (“Re-cinematography: improving the camera dynamics of casual video. In ACM Multimedia 2007 Conference Proceedings (2007), pp. 27-36”) stabilized the camcorder motion to be piecewise constant, which is similar with our method, but the ROI and the possibility of missing area completion were taken into consideration.
When filling up the missing image areas, there are some image inpainting approaches developed for recovering the missing holes in the image. Although these approaches can complete the missing areas with correct structure, but there will be obvious discontinuity if each video frame was recovered respectively. Litvin et al. (“Probabilistic Video Stabilization using Kalman Filtering and Mosaicking,” Proc. SPIE EI 2003, Vol. 5022, pp. 663-674, 2003) used mosaic method to fill up the missing areas in the stabilized video; however they did not consider the moving objects would appear at the boundary of the video. Wexler et al. (“Space-Time Video Completion,” Proc. IEEE CVPR 2004, Vol. 1, pp. 120-127, 2004) and Shiratori et al. (“Video Completion by Motion Field Transfer,” Proc. IEEE CVPR 2006, Vol. 1, pp. 411-418, 2006) filled up the missing holes by sampling the spatio-temporal volume patches from other portion of the video volume. The former approach used the most similar patch in color space for completing the missing holes and the later one used the patch with similar motion vector. The drawback of these methods is that they need large computing time for searching a proper patch. Jia et al. (“Video Repairing Inference of Foreground and Background under Severe Occlusion,” Proc. IEEE CVPR 2004, Vol. 1, pp. 364-371, 2004) and Patwardhan et al. (“Video Inpainting under Constrained Camera Motion,” IEEE TIP, Vol. 16, No. 2, pp. 545-553, 2007) segmented the video into two layers and recovered them individually. These methods focused on long and periodic observed time of the moving objects, but this is not guaranteed in common home videos.